Image processing apparatus and image processing method

ABSTRACT

A band processing circuit which generates image signals corresponding to different frequency bands from an image signal in which signals corresponding to different colors are arranged and which suppresses noise by synthesizing the image signals of the different frequency bands, a sampling circuit which generates image signal corresponding to the colors by sampling the image signal input from the band processing circuit in accordance with a predetermined arrangement, and a luminance/color generation circuit which generates a luminance signal in which aliasing is suppressed using an image signal output from the sampling circuit.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation of U.S. application Ser. No.13/025,053, filed Feb. 10, 2011, which claims priority InternationalPatent Application No. PCT/JP2010/052028, filed Feb. 12, 2010, which arehereby incorporated by reference herein in their entireties.

TECHNICAL FIELD

The present invention relates to image processing performed forsuppressing aliasing generated in an image signal.

BACKGROUND ART

Various types of color filter of an image pickup element such as a CCDsensor or a CMOS sensor have been used, and examples of such a colorfilter include a color filter having a combination of primary colors(red, green, and blue) and a color filter having a combination ofcomplementary colors (cyan, magenta, and yellow).

FIG. 13 is a diagram illustrating a primary-color Bayer arrangement ofan image pickup element. Red (R) and blue (B) are diagonally arranged ina pixel matrix of 2×2, green (G1 and G2) is diagonally arranged in theremaining two pixels, and this pattern is repeated.

When an object includes a high frequency component which exceedsresolution capability of the image pickup element, aliasing is generatedin an image signal generated by the image pickup element due to anadverse effect of the high frequency component. Therefore, variousmethods for suppressing aliasing have been proposed. For example, amethod using a combination of two luminance signals generated indifferent ways has been proposed in order to suppress generation ofaliasing.

One of the luminance signals is generated only using signalscorresponding to G (G1, G2) pixels without using signals correspondingto R and B pixels. First, values of signals other than signalscorresponding to the G pixels among the signals corresponding to the R,G, and B pixels obtained by digitalizing a signal output from the imagepickup element having the primary-color Bayer arrangement are set to 0.Next, a vertical lowpass filter (V-LPF) process which restricts a bandin a vertical direction and a horizontal lowpass filter (H-LPF) whichrestricts a band in a horizontal direction are performed. By this,signals of pixels which have been subjected to compensation using thesignals corresponding to the G pixels are generated and a luminancesignal of G is obtained. Hereinafter, a luminance signal obtained byperforming compensation on pixels which do not correspond to a certaincolor using signals corresponding to the certain color is referred to asa first luminance signal.

Alternatively, values of signals other than signals of the R pixels areset to 0 and similarly the V-LPF process and the H-LPF process areperformed to thereby generate a luminance signal of R. Similarly, valuesof signals other than signals of the B pixels are set to 0 and similarlythe V-LPF process and the H-LPF process are performed to therebygenerate a luminance signal of B. Then, the luminance signals of R and Bare added to the luminance signal of G, and a resultant signal may bereferred to as a first luminance signal.

The other luminance signal is generated using signals of all the colorsof the primary-color Bayer arrangement shown in FIG. 13. The V-LPFprocess which restricts a band in the vertical direction and the H-LPFwhich restricts a band in the horizontal direction are performed on thesignals corresponding to the pixels of all the R, G, and B colors whichare obtained by digitalizing the signals output from the image pickupelement having the primary-color Bayer arrangement withoutdistinguishing the colors so that a signal is newly obtained.Hereinafter, such a luminance signal obtained using the signals of allthe colors without distinguishing the colors is referred to as a secondluminance signal.

FIG. 14 is a diagram illustrating spatial frequency characteristics inwhich the first and second luminance signals can be resolved. An x axisdenotes a frequency space in a horizontal (H) direction of an object anda y axis denotes a frequency space in a vertical (V) direction of theobject. The further a point is located from an intersection between thex axis and the y axis, the higher a spatial frequency in the point is.

Resolution limits in the horizontal and vertical directions of the firstluminance signal generated only using the signals corresponding to the Gpixels are equal to a Nyquist frequency (π/2) of an arrangement of the Gpixels. However, since some diagonal lines do not include the G pixels,a limit resolution frequency in a diagonal direction is lower than thosein the horizontal and vertical directions and an inside portion of aregion 1401 having a diamond shape shown in FIG. 14 corresponds to aspatial frequency in which the first luminance signal can be resolved.Since, among the R, G, and B luminance signals, the G luminance signalobtained only using the signals corresponding to the G pixels has thehighest resolution, even when the first luminance signal is generated bysynthesizing the R, G, and B luminance signals with one another, thesame spatial frequency in which the first luminance signal can beresolved is obtained.

On the other hand, since the second luminance signal is generated usingthe signals corresponding to all the color pixels, when the object isachromatic, an outer square region 1402 shown in FIG. 14 corresponds toa spatial frequency in which the second luminance signal can beresolved. Unlike the first luminance signal, since any one of the colorpixels is included in all lines diagonally extending, a spatialfrequency in a diagonal direction in which the second luminance signalcan be resolved is higher than that of the first luminance signal.However, when a red object is captured, for example, signals output fromthe pixels other than the R pixels are negligible. Accordingly, only aresolution corresponding to a region 1403 which is a quarter of theregion corresponding to the achromatic object is obtained.

Taking characteristics of the first and second luminance signalsdescribed above into consideration, a configuration for suppressingaliasing included in an image signal by generating a luminance signalhas been proposed. For example, a configuration for generating aluminance signal by changing a mixing ratio of the first and secondluminance signals depending on a determination as to whether an objectis achromatic or chromatic has been proposed (refer to Patent Literature1). Furthermore, a configuration for generating a luminance signal bychanging a mixing ratio of the first and second luminance signalsdepending on a degree of the diagonal correlation of an object shown inFIG. 14 has been proposed (refer to Patent Literature 2).

However, although these methods are useful in terms of the suppressionof aliasing, noise signals other than the aliasing are not suppressed.For example, in recent years, miniaturization of pixels of image pickupelements has been developed. Therefore, noise may be increased due tothe miniaturization of pixels. Although various methods for suppressingsuch noise by performing signal processing have been proposed, imageblur is generated when such noise is suppressed, which is an adverseeffect.

To address this problem, a method for suppressing noise by dividing animage signal into a plurality of frequency components has been proposed(refer to Patent Literature 3). Furthermore, a method for suppressingnoise by generating an image signal by reducing an image signal andsynthesizing the reduced image signal with the original image signal hasbeen proposed (Patent Literature 4).

Specifically, a reduction process is performed on a signal of an inputimage so that a reduced image including a frequency component lower thanthat of the input image is generated. Then, edge strength is detectedusing the reduced image signal having the low frequency component, and aregion in which an edge component is to be maintained is obtained inaccordance with the edge strength. Weights of regions are changed so tatan image included in the region in which the edge component is to bemaintained is not blurred and the original image signal and the reducedimage signal having the low frequency component are synthesized witheach other to thereby newly generate an image signal.

CITATION LIST Patent Literature

-   PTL 1 Japanese Patent Laid-Open No. 2003-348609-   PTL 2 Japanese Patent Laid-Open No. 2008-072377-   PTL 3 Japanese Patent Laid-Open No. 2008-015741 PTL 4 Japanese    Patent Laid-Open No. 2009-199104

However, the method for suppressing noise by synthesizing image signalsof a plurality of frequency bands does not take aliasing intoconsideration.

Here, in a case of a sensor of a single plate such as the primary-colorBayer arrangement, when a downsampling process is performed at a time ofband division, all pixels may have color signals without performingcompensation on pixels of colors corresponding to signals of 0. When acertain row in the Bayer arrangement of the image pickup element istaken as an example, as shown in FIG. 15, color filters R, G, R, G, R,G, and so on are arranged in this order. A case where the downsamplingprocess is performed on half of pixels in the horizontal direction willbe described as an example. In this row, when only R signals are focusedon, the R signals originally arranged every other pixel are arranged inevery pixels through the downsampling process. Furthermore, also whenonly G signals are focused on, the G signals originally arranged everyother pixel are arranged in every pixels through the downsamplingprocess. Note that, in the downsampling process performed on the Gsignals, a value of a pixel corresponding to each of the G signals isobtained by calculating an average of values of adjacent pixels in orderto a position of the center of gravity of the G signal matches an Rsignal. As described above, by performing the downsampling process, allthe R, G, and B color signals are generated at each pixel positionwithout performing compensation on the pixels of the colorscorresponding to the signals of 0. Therefore, aliasing may be suppressedin such an image signal generated through the downsampling process. Notethat a process of generating signals of different colors for each pixelis referred to as synchronization.

An image signal generated through the downsampling process has beensubjected to the synchronization. Therefore, when the original imagesignal is to be subjected to the synchronization with the image signalwhich has been subjected to the downsampling process, the original imagesignal should be subjected to the synchronization. However, since asingle pixel corresponds to a single color in the original image signal,a compensation process should be performed on pixels of the individualcolors which correspond to a signal of 0 in order to make each of thepixels have signals of all the colors. Therefore, since the originalimage signals are subjected to the synchronization although the originalsignals have been subjected to the downsampling process in order tosuppress noise, an image signal obtained by synthesis include aliasing.

The present invention has been made in view of the problem describedabove and it is an object of the present invention to provide an imageprocessing apparatus which performs a noise process by dividing an inputsignal according to a plurality of bands and which is capable ofsuppressing aliasing in an image signal generated by sampling performedby an image pickup element.

SUMMARY OF INVENTION

According to an aspect of the present invention, an apparatus includes ageneration unit configured to receive an image signal in which signalscorresponding to a plurality of colors are arranged and generate aplurality of image signals corresponding to different frequency bandsusing the received image signal, a synthesis unit configured tosynthesize the plurality of image signals, a sampling unit configured togenerate image signals from an image signal obtained through thesynthesis by sampling the signals corresponding to the colors inaccordance with the arrangement, a first generation unit configured toreceive the image signals of the colors which have been subjected to thesampling and generate a first signal using an image signal obtained byperforming compensation on pixels which do not correspond to a firstcolor signal among pixels of the received image signal using the firstcolor signal, a second generation unit configured to receive the imagesignals corresponding to the color signals including the first colorsignal and generate a second signal using the color signals, and amixing unit configured to output a signal obtained by synthesizing thefirst and second signals or a signal obtained by selecting one of thefirst and second signals.

The present invention further provides an apparatus including areduction unit configured to reduce an image signal in which signalscorresponding to a plurality of colors are arranged so as to generate areduced image signal, a generation unit configured to receive thereduced image signal and generate a plurality of image signalscorresponding to different frequency bands from the image signal, asynthesis unit configured to synthesize the plurality of image signals,a first generation unit configured to receive the synthesized imagesignal and generate a first signal using an image signal obtained byperforming compensation on pixels which do not correspond to a firstcolor signal among the input image signals using the first color signal,a second generation unit configured to receive the synthesized imagesignal and generate a second signal using the plurality of colorsignals, a third generation unit configured to generate a third signalusing the image signal, and a mixing unit configured to output a signalobtained by synthesizing the first and second signals with each other ora signal obtained by selecting one of the first and second signals whena rate of a size of the reduced image signal to a size of the imagesignal in which the signals corresponding to the plurality of colors arearranged is larger than a threshold value, and output the third signalwhen the rate is equal to or smaller than the threshold value.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram schematically illustrating a configuration of adigital still camera according to an embodiment of the presentinvention.

FIG. 2 is a diagram schematically illustrating a portion of aconfiguration of an image processing circuit according to a firstembodiment of the present invention.

FIG. 3 is a flowchart illustrating a process performed by the imageprocessing circuit.

FIG. 4 is a diagram illustrating a synchronization process performed bya synchronization circuit.

FIG. 5 is a diagram illustrating filter coefficients in an edgedetection process performed by a synthesis ratio calculation circuit.

FIG. 6 is a diagram illustrating a mixing ratio α of ahigh-frequency-layer image signal in an edge component.

FIG. 7 is a diagram illustrating a process of generating a firstluminance signal Y_(A) performed by a first luminance signal generationcircuit.

FIG. 8 is a diagram illustrating a process of generating a secondluminance signal Y_(B) performed by a second luminance signal generationcircuit.

FIG. 9 is a diagram illustrating a mixing ratio β of the first luminancesignal Y_(A) in accordance with a saturation S.

FIG. 10 is a diagram schematically illustrating a portion of aconfiguration of an image processing circuit according to a secondembodiment of the present invention.

FIG. 11 is a diagram schematically illustrating a portion of aconfiguration of an image processing circuit according to a thirdembodiment of the present invention.

FIG. 12 is a flowchart illustrating a process performed by the imageprocessing circuit shown in FIG. 11.

FIG. 13 is a diagram illustrating a primary-color Bayer arrangement ofan image pickup element.

FIG. 14 is a diagram illustrating spatial frequency characteristics inwhich first and second luminance signals can be resolved.

FIG. 15 is a diagram illustrating a case where each pixel which has beensubjected to a downsampling process has all color signals.

DESCRIPTION OF EMBODIMENTS First Embodiment

FIG. 1 is a diagram schematically illustrating a configuration of adigital still camera according to an embodiment of the presentinvention. The present invention can be realized by not only a digitalstill camera but also a digital video camera, a personal computer whichincludes an application relating to image processing, or the like aslong as the image processing apparatus can perform image processing onan image signal.

In FIG. 1, an optical system 101 includes a lens unit including a zoomlens and a focus lens, an aperture device, and a shutter device. Theoptical system 101 controls magnification and a focus position of anobject image which is supplied to an image pickup element 102 and lightquantity. The image pickup element 102 is a photoelectric conversionelement such as a CCD (Charge Coupled Device) sensor or a CMOS(Complementary Metal Oxide Semiconductor) sensor and generates an imagesignal by converting the object image into an electric signal. In thisembodiment, the image pickup element 102 is constituted by a CCD sensorhaving a Bayer arrangement including color filters corresponding to R,G, and B.

A preprocessing circuit 103 includes a CDS (Correlated Double Sampling)circuit and an amplifying circuit. The CDS circuit suppresses darkcurrent included in an image signal generated by the image pickupelement 102 whereas the amplifying circuit amplifies an image signaloutput from the CDS circuit. An A/D converter 104 converts an imagesignal output from the preprocessing circuit 103 into a digital imagesignal.

An image processing circuit 105 performs a white balance process, anoise suppression process, a tone conversion process, and a contourcorrection process on the image signal so as to output the image signalas a luminance signal Y and color difference signals U and V.Furthermore, the image processing circuit 105 calculates a luminancevalue of the object and a focus value representing a focused state ofthe object using the image signal. The image processing circuit 105performs the image processing not only on an image signal output fromthe A/D converter 104 but also on an image signal read from a recordingmedium 109. A control circuit 106 controls the various circuits includedin the digital still camera of this embodiment so as to integrallycontrol operation of the digital still camera. The control circuit 106further controls driving of the optical system 101 and the image pickupelement 102 in accordance with the luminance value obtained from theimage signal which has been processed by the image processing circuit105 and an instruction supplied from an operation member 110.

A display memory 107 temporarily stores an image signal which is asource of an image to be displayed in a display device 108. The displaydevice 108 includes a liquid crystal display or an organic EL (ElectroLuminescence) display and displays an image using an image signalgenerated by the image pickup element 102 or an image signal read fromthe recording medium 109. The display device 108 serves as an electronicview finder by appropriately updating and displaying image signalsconsecutively read from the image pickup element 102. The display device108 may display not only an image but also text information such as adisplay state of the digital still camera, a shutter speed and anaperture value selected by a user or determined by the camera, andsensitivity information, and a graph representing distribution ofluminance measured by the image processing circuit 105. The recordingmedium 109 which stores image signals may be attachable to the digitalstill camera or may be incorporated in the digital still camera.

The operation member 110 is operated when the user supplies aninstruction to the digital still camera. A bus 111 is used totransmit/receive an image signal among the image processing circuit 105,the control circuit 106, the display memory 107, and the recordingmedium 109.

Next, an example of an operation of the digital still camera accordingto this embodiment at a time of image capturing will be described.

When the user operates the operation member 110 so as to supply aninstruction for starting preparation of image capturing, the controlcircuit 106 start controlling operations of the circuits. The imagepickup element 102 generates an analog image signal by photoelectricallyconverting an object image which is supplied through the optical system101. The A/D converter 104 digitizes the analog image signal which hasbeen processed by the preprocessing circuit 103. The image processingcircuit 105 performs the white balance process, the noise suppressionprocess, the tone conversion process, and the contour correctionprocess.

The image signal which has been processed by the image processingcircuit 105 is supplied through the display memory 107 to the displaydevice 108 which displays an image corresponding to the image signal. Asdescribed above, the display device 108 functions as an electronic viewfinder by updating the image of the object in real time using readconsecutive image signals and displaying the updated image.

This process is repeatedly performed until the user operates a shutterbutton included in the operation member 110. When the user operated theshutter button, the control circuit 106 controls the operation of theoptical system 101 again in accordance with a luminance value and afocus value obtained by the image processing circuit 105 and captures astill image. The image processing circuit 105 performs the various imageprocesses including the noise suppression process on an image signalcorresponding to the still image. Then, the recording medium 109 recordsthe image signal output from the image processing circuit 105. Note thatthe image processing circuit 105 performs the various image processesincluding the noise suppression process not only on still images butalso captured movies.

Here, the noise suppression process performed by the image processingcircuit 105 which is a characteristic of the present invention will bedescribed in detail. FIG. 2 is a diagram schematically illustrating aconfiguration of the image processing circuit 105.

The image processing circuit 105 of this embodiment includes a whitebalance circuit 201, a band processing circuit 202, a sampling circuit203, and a luminance/color generation circuit 204. The band processingcircuit 202 includes a synchronization circuit 205, a noise suppressioncircuit 206, a synthesis ratio calculation circuit 207, a reductioncircuit 208, a noise suppression circuit 209, an enlargement circuit210, and an image synthesis circuit 211. The luminance/color generationcircuit 204 includes a first luminance signal generation circuit 212, asecond luminance signal generation circuit 213, a luminance mixingcircuit 214, and a color generation circuit 215.

FIG. 3 is a flowchart illustrating a process performed by the whitebalance circuit 201, the band processing circuit 202, the samplingcircuit 203, and the luminance/color generation circuit 204 which areincluded in the image processing circuit 105. When receiving an imagesignal output from the A/D converter 104, the image processing circuit105 performs the process shown in the flowchart of FIG. 3.

In step S301, the white balance circuit 201 performs the white balanceprocess on an image signal supplied from the A/D converter 104. Here,the image signal is configured by a RAW format, and a single pixel hasone of signals corresponding to colors R, G, and B. The white balancecircuit 201 calculates gain coefficients for individual color signals sothat levels of the R, G, and B signals for a white object aresubstantially the same as one another. Note that the gain coefficientsare calculated using a general method, and therefore, a detaileddescription thereof is omitted.

In step S302, the synchronization circuit 205 performs a synchronizationprocess on the image signal output from the white balance circuit 201.FIG. 4 is a diagram illustrating the synchronization process performedby a synchronization circuit 205.

The image signal of the Bayer arrangement which has the RAW format andwhich is output from the white balance circuit 201 is supplied to thesynchronization circuit 205 which divides the image signal having theRAW format into image signals for individual colors. Specifically, thesynchronization circuit 205 generates an image signal corresponding to aG signal obtained by inserting 0 into signals of pixels other than Gpixels, an image signal corresponding to an R signal obtained byinserting 0 into signals of pixels other than R pixels, and an imagesignal corresponding to a B signal obtained by inserting 0 into signalsof pixels other than B pixels.

Subsequently, the synchronization circuit 205 performs thesynchronization process on the image signals corresponding to the G, R,and B signals so that each of pixels has the G, R, and B signals. Forexample, in the image signal corresponding to the R signal, assumingthat a signal at a coordinate (m, n) which has not been compensated foris represented by R(m, n) and the signal at the coordinate (m, n) whichhas been compensated for is represented by R_(P)(m, n), the signalR_(P)(m, n) which has been compensated for is calculated in accordancewith Expression (1) to Expression (4).

R _(P)(1,1)=R(1,1)  (1)

R _(P)(1,2)={R(1,1)+R(1,3)}/2  (2)

R _(P)(2,1)={R(1,1)+R(3,1)}/2  (3)

R _(P)(2,2)={R(1,1)+R(1,3)+R(3,1)+R(3,3)}/4  (4)

Similarly, in the image signal corresponding to the B signal, a signalB_(P)(m, n) which has been compensated for is calculated in the sameway.

Furthermore, in the image signal corresponding to the G signal, a signalG_(P)(m, n) which has been compensated for is calculated in accordancewith Expression (5) to Expression (8).

G _(P)(2,2)={G(1,2)+G(3,2)+G(2,1)+G(2,3)}/4  (5)

G _(P)(2,3)=G(2,3)  (6)

G _(P)(3,2)=G(3,2)  (7)

G _(P)(3,3)={G(2,3)+G(4,3)+G(3,2)+G(3,4)}/4  (8)

In step S303, the reduction circuit 208 receives an image signalcorresponding to the G_(P) signal, an image signal corresponding to theR_(P) signal, and an image signal corresponding to the B_(P) signal andgenerates an image signal of a lower band using these image signals.

The reduction circuit 208 performs the V-LPF process and the H-LPFprocess using a filter coefficient represented by [1, 2, 1] on the imagesignals corresponding to the G_(P) signal, the R_(P) signal, and theB_(P) signal. Thereafter, the reduction circuit 208 performs thedownsampling process on the image signals which have been subjected tothe LPF processes so that the numbers of pixels in the horizontal andvertical directions are reduced by half to thereby generate imagesignals corresponding to a G_(P1) signal, an R_(P1) signal, and a B_(P1)signal. The image signals corresponding to the G_(P) signal, the R_(P)signal, and the B_(P) signal which have not been subjected to thedownsampling process are referred to as high-frequency-layer imagesignals whereas the image signals corresponding to the G_(P1) signal,the R_(P1) signal, and the B_(P1) signal which have been subjected tothe downsampling process are referred to as low-frequency-layer imagesignals.

In this embodiment, the highest frequency band of thelow-frequency-layer image signals corresponds to half of the highestfrequency of the high-frequency-layer image signals. A lower frequencyband of the high-frequency-layer image signals overlaps with a frequencyband of the low-frequency-layer image signals.

Note that the reduction circuit 208 may generate the low-frequency-layerimage signals by reducing the number of the image signals of the Bayerarrangement which have the RAW format and which are output from thewhite balance circuit 201 and thereafter performing the LPF processes ona reduced number of image signals without using the synchronizationcircuit 205.

In step S304, the noise suppression circuit 206 performs the noisesuppression process on the high-frequency-layer image signals whereasthe noise suppression circuit 209 performs the noise suppression processon the low-frequency-layer image signals. The processes performed by thenoise suppression circuits 206 and 209 are the same as each other, andtherefore, only the noise suppression process performed by the noisesuppression circuit 206 will be described as an example.

The noise suppression circuit 206 performs the noise suppression processusing target signals of the individual color signals and surroundingpixels including ranges of matrices of 5×5 pixels with the target pixelsas centers.

When an image signal corresponding to the G_(P) signal is taken as anexample, the noise suppression circuit 206 calculates absolute values ofdifferences between a signal level of a target pixel and signal levelsof surrounding pixels and compares resultant values with a thresholdvalue. Specifically, assuming that the signal level of the target pixelis denoted by G_(P)(s, t), the signal levels of the surrounding pixelsare denoted by G_(P)(i, j), and the threshold value is denoted by TH,the signal levels G_(P)(i, j) which satisfy Expression (9) is obtained.

|G _(P)(i,j)−G _(P)(s,t)|<TH (note: s−2≦i≦s+2,t−2≦j≦t+2)  (9)

Then, the signal levels G_(P)(i, j) which satisfy Expression (9) areextracted, a value of the luminance signal (s, t) of the target pixel isreplaced by an average value of the signal levels G_(P)(i, j). The noisesuppression circuit 206 performs similar processes on thehigh-frequency-layer image signals corresponding to the R_(P) signal andthe B_(P) signal.

The noise suppression circuit 206 suppresses noise of thehigh-frequency-layer image signals by performing the processes performedon the G_(P) signal, the R_(P) signal, and the B_(P) signal on all thepixels. Note that since the range of a matrix of 5×5 is not allowed tobe set in edge portions of the image signal, a method for settingsurrounding pixels is appropriately changed for such a pixel in order toperform the noise suppression process.

The noise suppression circuit 209 also suppresses noise of thelow-frequency-layer image signals by performing the similar process onthe G_(P1) signal, the R_(P1) signal, and the B_(P1) signalcorresponding to the low-frequency-layer image signals. It is apparentthat the method for suppressing noise is not limited to this, andvarious general methods for suppressing noise may be employed.

Then, the low-frequency-layer image signals which have been subjected tothe noise suppression process by the noise suppression circuit 209 areoutput to the enlargement circuit 210. The enlargement circuit 210performs an upsampling process on the low-frequency-layer image signalswhich have been subjected to the noise suppression process so that thenumber of pixels of each of the low-frequency-layer image signalsbecomes equal to the number of pixels of a corresponding one of thehigh-frequency-layer image signals. Specifically, the number of pixelsof each of the low-frequency-layer image signals is increased by doubleand signals of pixels newly generated are set to 0. Thereafter, theenlargement circuit 210 performs linear compensation on the pixelshaving the values of 0 using values of surrounding pixels so that thepixels at all positions have signals. These processes are separatelyperformed on the G_(P1) signals, the R_(P1) signals, and the B_(P1)signals.

In step S305, the synthesis ratio calculation circuit 207 calculates aratio of synthesis of the high-frequency-layer image signals and thelow-frequency-layer image signals. Specifically, the synthesis ratiocalculation circuit 207 obtains an edge component of each of the pixelsusing a filter shown in FIG. 5 in terms of the G_(P) signalscorresponding to the high-frequency-layer image signals. Then, thesynthesis ratio calculation circuit 207 reads a mixing ratio αcorresponding to the edge component from a memory not shown. FIG. 6 is adiagram illustrating the mixing ratio α of the high-frequency-layerimage signals in accordance with the edge component. As shown in FIG. 6,in the synthesis ratio calculation circuit 207, a pixel having a highedge component has a high mixing ratio α so that the mixing ratio of thehigh-frequency-layer image signals is high whereas a pixel having a lowedge component has a low mixing ratio α so that the mixing ratio of thelow-frequency-layer image signals is low.

In step S306, the image synthesis circuit 211 synthesizes the G_(P)signal corresponding to the high-frequency-layer image signal with theG_(P1) signal corresponding to the low-frequency-layer image signalusing the mixing ratio α obtained by the synthesis ratio calculationcircuit 207 so as to newly obtain a G signal. Specifically, in each ofthe pixels, the G_(P) signal corresponding to the high-frequency-layerimage signal which has been subjected to the noise suppression processby the noise suppression circuit 206 and the G_(P1) signal correspondingto the low-frequency-layer image signal which has been subjected to theenlargement process by the enlargement circuit 210 are added to eachother in accordance with Expression (10).

G=α×G _(P)+(1−α)×G _(P1)  (10)

Similarly, in each of the pixels, the R_(P) signal and the B_(P) signalcorresponding to the high-frequency-layer image signals which have beensubjected to the noise suppression process by the noise suppressioncircuit 206 and the R_(P1) signal and the B_(P1) signal corresponding tothe low-frequency-layer image signals which have been subjected to theenlargement process by the enlargement circuit 210 are added to eachother in accordance with Expression (11) and Expression (12),respectively.

R=α×R _(P)+(1−α)×R _(P1)  (11)

B=α×B _(P)+(1−α)×B _(P1)  (12)

Then, the G, R, and B signals which are obtained after the additions andwhich are output from the image synthesis circuit 211 are supplied tothe sampling circuit 203.

In step S307, the sampling circuit 203 performs sampling on the G, R,and B signals in accordance with the Bayer arrangement so as to generatea single image signal in accordance with the Bayer arrangement.Specifically, the sampling circuit 203 generates an image signal havinga pattern in which a matrix of 2×2 signals, i.e., four signals,configured such that an R signal and a B signal are diagonally arrangedin two pixels and G signals are diagonally arranged in remaining twopixels is repeated. Then, the sampling circuit 203 supplies the imagesignal which is obtained by sampling and which is regenerated inaccordance with the Bayer arrangement to the first luminance signalgeneration circuit 212, the second luminance signal generation circuit213, and the color generation circuit 215.

Subsequently, in step S308, the first luminance signal generationcircuit 212 generates a first luminance signal Y_(A). FIG. 7 is adiagram illustrating a process of generating the first luminance signalY_(A) performed by the first luminance signal generation circuit 212.

The first luminance signal generation circuit 212, as shown in FIG. 7,divides the image signal supplied from the sampling circuit 203 intoimage signals of the individual colors and performs a compensationprocess so as to generate color signals in individual pixels whereby asynchronization process is performed.

For example, in an image signal corresponding to an R signal, assumingthat a signal in a coordinate (m, n) which has not been subjected to thecompensation process is denoted by R(m, n) and a signal in thecoordinate (m, n) which has been subjected to the compensation processis denoted by R_(A)(m, n), the signal R_(A)(m, n) is calculated inaccordance with Expression (13) to Expression (16).

RA(1,1)=R(1,1)  (13)

RA(1,2)={R(1,1)+R(1,3)}/2  (14)

R _(A)(2,1)={R(1,1)+R(3,1)}/2  (15)

R _(A)(2,2)={R(1,1)+R(1,3)+R(3,1)+R(3,3)}/4  (16)

Similarly, in an image signal corresponding to a B signal, a signalB_(A)(m, n) which has been subjected to the compensation process iscalculated in the same way.

Furthermore, in an image signal corresponding to a G signal, a signalG_(A)(m, n) which has been subjected to the compensation process iscalculated in accordance with Expression (17) to Expression (20).

Ga(2,2)={G(1,2)+G(3,2)+G(2,1)+G(2,3)}/4  (17)

Ga(2,3)={4×G(2,3)+G(1,2)+G(1,4)+G(3,2)+G(3,4)}/8  (18)

Ga(3,2)={4×G(3,2)+G(2,1)+G(2,3)+G(4,1)+G(4,3)}/8  (19)

Ga(3,3)={G(2,3)+G(4,3)+G(3,2)+G(3,4)}/4  (20)

It is apparent that the compensation methods are merely examples, andvarious other general compensation methods may be employed. For example,in order to suppress degradation of a high band characteristic of anoriginal signal, the correlations of the object for individualdirections may be determined in advance and heavy weight may be appliedon signals located in a direction corresponding to the highestcorrelation whereby compensation is performed. Specifically, thecorrelation between signals of G pixels which are vertically adjacent toa pixel to be compensated for may be compared with the correlationbetween signals of G pixels which are horizontally adjacent to the pixelto be compensated for. Then, a rate of the signals of the G pixelshaving the higher correlation may be made large so that a signalG_(A)(m, n) of the pixel to be compensated for is obtained.Alternatively, the signal G_(A)(m, n) of the pixel to be compensated formay be obtained only using the signals of the G pixels corresponding tothe higher correlation.

In step S302, the synchronization circuit 205 does not accurately obtainthe correlations of the pixels located near the target pixel since noisecomponents are superposed on the pixels shown in FIG. 13. Therefore, asignal of the pixel to be compensated for is obtained by simple linearcompensation. On the other hand, the first luminance signal generationcircuit 212 performs the compensation process using signals in whichnoise components are suppressed by the band processing circuit 202.Therefore, the first luminance signal generation circuit 212 performsthe compensation process in accordance with the correlations of thepixels located in the vicinity of the pixel to be compensated for.

The R_(A) signal, the G_(A) signal, the B_(A) signal obtained asdescribed above are assigned to Expression (21) so that a firstluminance signal Y_(A)(m, n) of a pixel of a coordinate (m, n) isobtained.

Y _(A)(m,n)=0.3×R _(A)(m,n)+0.6×G _(A)(m,n)+0.1×B _(A)(m,n)  (21)

Then, the first luminance signal generation circuit 212 outputs theobtained first luminance signal Y_(A) to the luminance mixing circuit214.

Note that, although the case where the R_(A) signal, the G_(A) signal,and the B_(A) signal are used to obtain the first luminance signal Y_(A)is taken as an example, the G_(A) signal may be used as the firstluminance signal Y_(A). Any luminance signal may be used as the firstluminance signal Y_(A) as long as the first luminance signal Y_(A) isobtained by performing the compensation process on an image signalcorresponding to the G signal which is a first color signal.

In step S309, the second luminance signal generation circuit 213generates a second luminance signal Y_(B). Unlike the first luminancesignal Y_(A), the second luminance signal Y_(B) is not generated bydistinguishing color signals but is generated by processing signalscorresponding to the pixels of all the colors. FIG. 8 is a diagramillustrating a process of generating the second luminance signal Y_(B)performed by the second luminance signal generation circuit 213.

As shown in FIG. 8, the second luminance signal generation circuit 213receives the image signal output from the sampling circuit 203. Thesecond luminance signal generation circuit 213 performs the verticallowpass filter (V-LPF) process and the horizontal lowpass filter (H-LPF)process so as to generate the second luminance signal Y_(B). In theV-LPF process and the H-LPF process, a filter coefficient represented by[1, 2, 1], for example, may be used. Alternatively, a direction of thefilter or the filter coefficient may be changed in accordance with astate of an edge of the image signal or a level of the correlation withthe surrounding pixels.

Note that the process to be performed by the second luminance signalgeneration circuit 213 may be eliminated and the image signal of the RAWformat may be used as the second luminance signal Y₅. That is, secondluminance signals Y_(B) for individual pixels can be obtained inaccordance with Expression (22) to Expression (25).

Y _(B)(1,1)=R(1,1)  (22)

Y _(B)(1,2)=G(1,2)  (23)

Y _(B)(2,1)=G(2,1)  (24)

Y _(B)(2,2)=B(2,2)  (25)

In step S310, the luminance mixing circuit 214 mixes the first luminancesignal Y_(A) and the second luminance signal Y_(B) so as to generate aluminance signal Y. Assuming that a mixing ratio of the first luminancesignal in the luminance signal Y is denoted by β, the luminance mixingcircuit 214 obtains luminance signals for individual pixels by assigningthe first luminance signal Y_(A) and the second luminance signal Y_(B)in Expression (26).

Y=β×Y _(A)+(1−β)×Y _(B)  (26)

Here, in this embodiment, the luminance mixing circuit 214 determinesthe mixing ratio β of the first luminance signal Y_(A) in the luminancesignal Y in accordance with a saturation S of the object. A method forobtaining the mixing ratio β will be described. As with the firstluminance signal generation circuit 212, the luminance mixing circuit214 divides the image signal of the RAW format into image signals of theindividual colors and performs the synchronization. Then, an absolutevalue of a difference between an R signal and a G signal of each ofpixels is added to an absolute value of a difference between a B signaland the G signal of the pixel so that a saturation S of each of thepixels is obtained.

S=|R−G|+|B−G|  (27)

The luminance mixing circuit 214 reads the mixing ratio β correspondingto the saturation S from the memory not shown. FIG. 9 is a diagramillustrating the mixing ratio β of the first luminance signal Y_(A) inaccordance with the saturation S. As shown in FIG. 9, the luminancemixing circuit 214 sets the mixing ratio β to be large, that is, theluminance mixing circuit 214 sets a mixing ratio of the first luminancesignal Y_(A) to be large in a case of a pixel having a high saturationwhereas the luminance mixing circuit 214 sets the mixing ratio β to besmall, that is, the luminance mixing circuit 214 sets a mixing ratio ofthe second luminance signal Y_(A) to be large in a case of a pixelhaving a low saturation.

Note that the method for obtaining a mixing ratio of the first luminancesignal Y_(A) to the second luminance signal Y_(B) is not limited tothis, and a final mixing ratio may be determined by multiplying a mixingratio obtained using an absolute value of a difference between an Rsignal and a G signal by a mixing ratio obtained using an absolute valueof a difference between a B signal and the G signal. Furthermore,instead of the process of adding and synthesizing the first luminancesignal Y_(A) and the second luminance signal Y_(B) with each other, oneof the first luminance signal Y_(A) and the second luminance signalY_(B) which corresponds to a higher mixing ratio may be selected.Alternatively, in a case where a region in which it is determined thatthe diagonal correlation is higher than the threshold value is detected,the second luminance signal Y_(B) may be used only for the region.Furthermore, since a configuration ratio of colors included in a highfrequency component of the first luminance signal Y_(A) is differentfrom that of the second luminance signal Y_(B), when a certain color ofthe object is strong, a value of the first luminance signal Y_(A) and avalue of the second luminance signal Y_(B) is considerably differentfrom each other. Therefore, a low frequency component of the firstluminance signal Y_(A) and a high frequency component of the secondluminance signal Y_(B) may be obtained and synthesized with each otherso that a luminance signal is obtained, and the obtained luminancesignal and the first luminance signal Y_(A) may be mixed with each otherwith the mixing ratio described above.

In step S311, the color generation circuit 215 generates colordifference signals U and V using the R, G, and B signals output from thesampling circuit 203. The color generation circuit 215 performs a colorcompensation process, a fake color removing process, a matrix conversionprocess, and the like on the R, G, and B signals so as to generate thecolor difference signals U and V. The color difference signals U and Vmay be generated by a general method, and therefore, a detaileddescription of the method is omitted.

By performing the processes, the luminance signal Y in which thealiasing is suppressed and the color difference signals U and V aregenerated in the luminance/color generation circuit 204.

As described above, according to the configuration of this embodiment,the band processing circuit 202 executes the noise suppression processby dividing an image signal according to a plurality of bands, and theluminance/color generation circuit 204 executes the process ofsuppressing the aliasing.

Note that although the case where only low-frequency-layer image signalsin a single layer is generated has been taken as an example, the presentinvention is not limited to this. A plurality of reduction circuits 208,a plurality of noise suppression circuits 209, and a plurality ofenlargement circuits 210 may be provided, the noise suppression processmay be performed on individual low-frequency-layer image signals whichare multilayered according to frequency bands, and thelow-frequency-layer image signals may be synthesized with one another bythe image synthesis circuit 211.

Furthermore, a method of the noise suppression process performed on thehigh-frequency-layer image signals and a method of the noise suppressionprocess performed on the low-frequency-layer image signals may bedifferent from each other. For example, the high-frequency-layer imagesignals may be subjected to the noise suppression process in the waydescribed in step S304 whereas the low-frequency-layer image signals maybe subjected to a noise suppression process including the V-LPF processand the H-LPF process.

Alternatively, if the reduction circuit 208 has performed a noisesuppression process such as the LPF processes when the reduction circuit208 generates low-frequency-layer image signals, the noise suppressioncircuit 209 which processes low-frequency-layer image signals may beeliminated.

Second Embodiment

Next, a second embodiment of the present invention will be described. Inthe first embodiment, an image signal having a frequency band lower thanthat of an original image signal is generated by the band processingcircuit 202. However, in this embodiment, an original image signal isdivided into a plurality of frequency components corresponding todifferent frequency bands which do not overlap with one another.

FIG. 10 is a diagram schematically illustrating another configuration ofthe image processing circuit 105 according to a second embodiment of thepresent invention. In FIG. 10, circuits having configurations the sameas those shown in FIG. 2 are denoted by reference numerals the same asthose shown in FIG. 2. The image processing circuit 105 of thisembodiment includes a white balance circuit 201, a band processingcircuit 1001, a sampling circuit 203, and a luminance/color generationcircuit 204.

A synchronization circuit 1002 included in the band processing circuit1001 performs the same synchronization processing as the synchronizationcircuit 205 on an image signal output from the white balance circuit 201so as to generate an image signal corresponding to a G_(P) signal, animage signal corresponding to an R_(P) signal, and an image signalcorresponding to a B_(P) signal. In each of image signals, each ofpixels output from the synchronization circuit 1002 includes the G_(P)signal, the R_(P) signal, and the B_(P) signal.

The image signals corresponding to the G_(P) signal, the R_(P) signal,and the B_(P) signal are supplied to a lowpass filter (LPF) 1003 and ahighpass filter (HPF) 1004.

As with the reduction circuit 208 shown in FIG. 2, the lowpass filter1003 performs a V-LPF process and an H-LPF process on the G_(P) signal,the R_(P) signal, and the B_(P) signal so as to generate a G_(PL)signal, an R_(PL) signal, and a B_(PL) signal.

The highpass filter 1004 subtracts the G_(PL) signal, the R_(PL) signal,and the B_(PL) signal which are generated by the lowpass filter 1003from the G_(P) signal, the R_(P) signal, and the B_(P) signal which areoutput from the synchronization circuit 1002. As a result, the highpassfilter 1004 extracts a G_(PH) signal, an R_(PH) signal, and a B_(PH)signal which are high frequency components of the G_(P) signal, theR_(P) signal, and the B_(P) signal, respectively.

The G_(PH) signal, the R_(PH) signal, and the B_(PH) signal are suppliedto a noise suppression circuit 1005 which performs the same noisesuppression process as the noise suppression circuit 206 shown in FIG.2. Furthermore, the G_(PH) signal, the R_(PH) signal, and the B_(PH)signal are supplied to a synthesis ratio calculation circuit 1006 wherea synthesis ratio is calculated in the same way as the synthesis ratiocalculation circuit 207 shown in FIG. 2.

The G_(PL) signal, the R_(PL) signal, and the B_(PL) signal which aregenerated by the lowpass filter 1003 are supplied to a downsampling (DS)circuit 1007. The DS circuit 1007 performs a downsampling process on theG_(PL) signal, the R_(PL) signal, and the B_(PL) signal so that thenumbers of pixels in horizontal and vertical directions are reduced byhalf whereby a G_(P1) signal, a P_(R1) signal, and a B_(P1) signal aregenerated.

The G_(P1) signal, the R_(P1) signal, and the B_(P1) signal output fromthe DS circuit 1007 are supplied to an LPF 1008 and an HPF 1009.

As with the reduction circuit 208 shown in FIG. 2, the LPF 1008 performsthe V-LPF process and the H-LPF process on the G_(P1) signal, the R_(P1)signal, and the B_(P1) signal so as to generate a G_(PL1) signal, aR_(PL1) signal, and a B_(PL1) signal.

The HPF 1009 subtracts the G_(PL1) signal, the R_(PL1) signal, and theB_(PL1) signal which are generated by the LPF 1008 from the G_(P1)signal, the R_(P1) signal, and the B_(P1) signal which are output fromthe DS circuit 1007. As a result, the HPF 1009 extracts a G_(PH1)signal, an R_(PL1) signal, and a B_(PH1) signal which are high frequencycomponents of the G_(P1) signal, the R_(P1) signal, and the B_(P1)signal.

The G_(PH1) signal, the R_(PH1) signal, and the B_(PH1) signal aresupplied to a noise suppression circuit 1010 where the same noisesuppression process as the noise suppression circuit 206 shown in FIG. 2is performed. Furthermore, the G_(PH1) signal, the R_(PH1) signal, andthe B_(PH1) signal are supplied to a synthesis ratio calculation circuit1011 where a synthesis ratio is calculated in the same way as thesynthesis ratio calculation circuit 207 shown in FIG. 2.

The G_(PL1) signal, the R_(PL1) signal, and the B_(PL1) signal which aregenerated by the LPF 1008 are supplied to a DS circuit 1012. The DScircuit 1012 performs a downsampling process on the G_(PL1) signal, theR_(PL1) signal, and the B_(PL1) signal so that the numbers of pixels inhorizontal and vertical directions are reduced by half whereby a G_(P2)signal, a P_(R2) signal, and a B_(P2) signal are generated.

The G_(P2) signal, the R_(P2) signal, the B_(P2) signal which are outputfrom the DS circuit 1012 are supplied to an LPF 1013 and an HPF 1014. Aswith the reduction circuit 208 shown in FIG. 2, the LPF 1013 performsthe V-LPF process and the H-LPF process on the G_(P2) signal, the R_(P2)signal, and the B_(P2) signal so as to generate a G_(PL2) signal, anR_(PL2) signal, and a B_(PL2) signal.

The HPF 1014 subtracts the G_(PL2) signal, the R_(PL2) signal, and theB_(PL2) signal which are generated by the LPF 1013 from the G_(P2)signal, the R_(P2) signal, and the B_(P2) signal which are output fromthe DS circuit 1012. As a result, the HPF 1014 extracts a G_(PH2)signal, an R_(PH2) signal, and a B_(PH2) signal which are high frequencycomponents of the G_(P2) signal, the R_(P2) signal, and the B_(P2)signal.

The G_(PH2) signal, the R_(PH2) signal, and the B_(PH2) signal aresupplied to a noise suppression circuit 1015 where the same noisesuppression process as the noise suppression circuit 206 shown in FIG. 2is performed. Furthermore, the G_(PH2) signal, the R_(PH2) signal, andthe B_(PH2) signal are supplied to a synthesis ratio calculation circuit1016 where a synthesis ratio is calculated in the same way as thesynthesis ratio calculation circuit 207 shown in FIG. 2.

The G_(PL2) signal, the R_(PL2) signal, and the B_(PL2) signal which aregenerated by the LPF 1013 are supplied to a DS circuit 1017. The DScircuit 1017 performs a downsampling process on the G_(PL2) signal, theR_(PL2) signal, and the B_(PL2) signal so that the numbers of pixels inhorizontal and vertical directions are reduced by half whereby a G_(P3)signal, a P_(R3) signal, and a B_(P3) signal are generated.

The G_(P3) signal, the R_(P3) signal, and the B_(P3) signal are suppliedto a noise suppression circuit 1018 where the same noise suppressionprocess as the noise suppression circuit 206 shown in FIG. 2 isperformed.

Frequency bands of the G_(PH) signal, the G_(PH1) signal, the G_(PH2)signal, and the G_(P3) signal are not superposed with one another, andthe frequency bands are lowered from the G_(PH) signal to the G_(P3)signal. Frequency bands of the R_(PH) signal to the R_(P3) signal andfrequency bands of the B_(PH) signal to the B_(P3) signal are similarlylowered.

An enlargement circuit 1019 performs upsampling process on theindividual signals supplied from the noise suppression circuit 1018 sothat the numbers of pixels corresponding to the signals supplied fromthe noise suppression circuit 1018 are equal to the numbers of pixelsoutput from the noise suppression circuit 1015. An image synthesiscircuit 1020 synthesizes the signals output from the enlargement circuit1019 with the signals output from the noise suppression circuit 1015 forindividual colors by the same method as the image synthesis circuit 211shown in FIG. 2 using the synthesis ratio obtained by the synthesisratio calculation circuit 1016.

An enlargement circuit 1021 performs an upsampling process on thesignals output from the image synthesis circuit 1020 so that the numbersof pixels corresponding to the signals output from the image synthesiscircuit 1020 are equal to the numbers of pixels output from the noisesuppression circuit 1010. An image synthesis circuit 1022 synthesizesthe signals output from the enlargement circuit 1021 with the signalsoutput from the noise suppression circuit 1010 for individual colors bythe same method as the image synthesis circuit 211 shown in FIG. 2 usingthe synthesis ratio obtained by the synthesis ratio calculation circuit1011.

Furthermore, an enlargement circuit 1023 performs an upsampling processon the signals output from the image synthesis circuit 1022 so that thenumbers of pixels corresponding to the signals output from the imagesynthesis circuit 1022 are equal to the numbers of pixels output fromthe noise suppression circuit 1005. An image synthesis circuit 1024synthesizes the signals output from the enlargement circuit 1023 withthe signals output from the noise suppression circuit 1005 forindividual colors by the same method as the image synthesis circuit 211shown in FIG. 2 using the synthesis ratio obtained by the synthesisratio calculation circuit 1006.

That is, the processes performed by the LPFs 1003, 1008, and 1013, andthe DS circuits 1007, 1012, and 1017 correspond to the process performedin step S303 shown in FIG. 3, and the processes performed by the noisesuppression circuits 1005, 1010, 1015, and 1018 correspond to theprocess performed in step S304 shown in FIG. 3. Furthermore, theprocesses performed by the synthesis ratio calculation circuits 1006,1011, and 1016 correspond to the process performed in step S305 shown inFIG. 3, and the processes performed by the enlargement circuits 1019,1021, and 1023 and the image synthesis circuits 1020, 1022, and 1024correspond to the process performed in step S306 shown in FIG. 3.

Then, the image signals corresponding to the G signal, the R signal, andthe B signal output from the image synthesis circuit 1024 are subjectedto a sampling process performed by the sampling circuit 203 inaccordance with the Bayer arrangement, and the image signals which havebeen subjected to the sampling process are supplied to theluminance/color generation circuit 204.

As described above, even when the band processing circuit is configuredsuch that the noise suppression process is performed after the originalimage signal is divided into a plurality of frequency componentscorresponding to different frequency bands which do not overlap with oneanother, the same advantage as the first embodiment can be attained.

Third Embodiment

Next, a third embodiment of the present invention will be described.This embodiment is different from the first embodiment in that a size ofan image signal output from an image processing circuit 105 can bechanged (resized) and different methods for generating a luminancesignal Y are used depending on sizes of the output image signal.

FIG. 11 is a diagram schematically illustrating a portion of aconfiguration of the image processing circuit 105. In FIG. 11, circuitshaving configurations the same as those shown in FIG. 2 are denoted byreference numerals the same as those shown in FIG. 2. The imageprocessing circuit 105 in this embodiment includes a white balancecircuit 201, a band processing circuit 1101, a sampling circuit 1102,and a luminance/color generation circuit 1103.

The band processing circuit 1101 includes a synchronization/reductioncircuit 1104 instead of the synchronization circuit 205 shown in FIG. 2.In the first and second embodiments, the number of pixels correspondingto an image signal input to the image processing circuit 105 is equal tothe number of pixels corresponding to an image signal output from theimage processing circuit 105. However, in this embodiment, the number ofpixels of an image signal output from the image processing circuit 105may be reduced relative to the number of pixels of an image signal inputto the image processing circuit 105 using the synchronization/reductioncircuit 1104. A user may specify a size of an image signal to be outputusing an operation member 110. A control circuit 106 instructs thesynchronization/reduction circuit 1104 to change the size of the imagesignal output from the image processing circuit 105 in accordance withthe size specified by the user.

Furthermore, the sampling circuit 1102 determines whether a process ofsampling a G signal, an R signal, and a B signal from the image signaloutput from the image synthesis circuit 211 in accordance with the Bayerarrangement, i.e., a re-Bayer arrangement process, is to be performed inaccordance with operation of the synchronization/reduction circuit 1104.

Furthermore, the luminance/color generation circuit 1103 includes athird luminance signal generation circuit 1105 and a luminance mixingcircuit 1106 instead of the luminance mixing circuit 214 shown in FIG.2. The luminance/color generation circuit 1103 switches the methods forgenerating a luminance signal Y from one to another in accordance withan operation of the synchronization/reduction circuit 1104.

FIG. 12 is a flowchart illustrating a process performed by the whitebalance circuit 201, the band processing circuit 1101, the samplingcircuit 1102, and the luminance/color generation circuit 1103 includedin the image processing circuit 105. When receiving an image signaloutput from the A/D converter 104, the image processing circuit 105performs the process of the flowchart shown in FIG. 12. Note that, inthe flowchart shown in FIG. 12, processes the same as those shown inFIG. 3 are denoted by step numbers the same as those shown in FIG. 3.

In step S301, the white balance circuit 201 performs a white balanceprocess on an image signal supplied from the A/D converter 104.

In step S1201, the synchronization/reduction circuit 1104 generates areduced image signal from the image signal output from the white balancecircuit 201 in accordance with a size of the image signal specified bythe control circuit 106.

In step S1202, the synchronization/reduction circuit 1104 determineswhether a rate of the size of the image signal specified by the controlcircuit 106 to a size of the image signal of a RAW format input to theimage processing circuit 105 is equal to or smaller than a thresholdvalue. In this embodiment, the threshold value is ½ since an imagesignal obtained from an image pickup element 102 having the Bayerarrangement is employed. When the synchronization/reduction circuit 1104determines that the size of the image signal specified by the controlcircuit 106 is half of the size of the image signal input to the imageprocessing circuit 105, the process proceeds to step S1202.

Note that the fact that the size of the image signal is reduced by halfmeans that the numbers of pixels in a vertical and horizontal directionsare reduced by half. If a rate of the size of the reduced image signalto the size of the image signal input to the image processing circuit105 is equal to or smaller than ½, even when the reduced image signal isdivided into image signals for individual colors, pixels having values 0are not generated in the image signals of the individual colors, andaccordingly, aliasing can be suppressed.

In step S1203, a reduction circuit 208 receives the reduced image signaloutput from the synchronization/reduction circuit 1104 and generates alow-frequency-layer image signal using the reduced image signal by thesame method as step S303 shown in FIG. 3.

In step S1204, a noise suppression circuit 206 performs the same noisesuppression method as step S304 shown in FIG. 3 on the image signaloutput from the synchronization/reduction circuit 1104, that is, ahigh-frequency-layer image signal.

In step S1205, the synthesis ratio calculation circuit 207 calculates asynthesis ratio of the high-frequency-layer image signal and thelow-frequency-layer image signal using a method the same as that in stepS305 shown in FIG. 3.

In step S1206, the image synthesis circuit 211 synthesizes thehigh-frequency-layer image signal with the low-frequency-layer imagesignal using a method the same as that in step S306 shown in FIG. 3 soas to obtain image signals constituted by a new G signal, a new Rsignal, and a new B signal. The image signals generated by the imagesynthesis circuit 211 are supplied to the sampling circuit 1102.

Note that, when the size of the reduced image signal is equal to orsmaller than half the size of the image signal input to the imageprocessing circuit 105, the sampling circuit 1102 does not perform asampling process and outputs the image signals without change to thethird luminance signal generation circuit 1105.

In step S1207, the third luminance signal generation circuit 1105generates a luminance signal Y in accordance with Expression (21) of thefirst embodiment. Since the image signals input to the third luminancesignal generation circuit 1105 have been subjected to a synchronizationprocess, unlike the first luminance signal generation circuit 212, thethird luminance signal generation circuit 1105 is not required toperform another synchronization process.

Then, the luminance mixing circuit 1106 outputs a luminance signalgenerated by the third luminance signal generation circuit 1105 as theluminance signal Y without change.

In step S1208, the color generation circuit 215 generates colordifference signals U and V by the same method as step S311 shown in FIG.3 and outputs the color difference signals U and V.

In this way, when the rate of the size of the reduced image signal tothe size of the image signal input to the image processing circuit 105is equal to or smaller than ½, the reduced image signal output from thesynchronization/reduction circuit 1104 serves as an image signal whichhas been synchronized and in which an adverse effect of aliasing issuppressed. Specifically, even when the reduced image signal is dividedinto a plurality of image signals corresponding to different frequencybands and the image signals are synthesized with one another, theadverse effect of aliasing is not increased. Accordingly, when the thirdluminance signal generation circuit 1105 generates the luminance signalY using the image signal output from the image synthesis circuit 211, aluminance signal in which the adverse effect of aliasing is suppressedmay be obtained without mixing a plurality of luminance signalsgenerated by different methods.

Therefore, when the size of the reduced image signal is equal to orsmaller than half the size of the image signal input to the imageprocessing circuit 105, the sampling circuit 1102 does not perform asampling process and outputs the input image signal to the thirdluminance signal generation circuit 1105 without change.

Referring back to step S1202, when the size of the reduced image signalis not equal to or smaller than the size of the image signal input tothe image processing circuit 105, the synchronization/reduction circuit1104 proceeds to step S302.

Then, the digital still camera performs the same processes as step S302to step S311 shown in FIG. 3. Note that the synchronization/reductioncircuit 1104 performs the same process as the synchronization circuit205 shown in FIG. 2, the sampling circuit 1102 performs the same processas the sampling circuit 203 shown in FIG. 2, and the luminance mixingcircuit 1106 performs the same process as the luminance mixing circuit214 shown in FIG. 2.

As described above, in the digital still camera according to thisembodiment, if the image processing circuit 105 can reduce a size of animage signal to the degree that the adverse effect of aliasing issuppressed, the image processing circuit 105 does not perform a processof synthesizing a plurality of luminance signals generated by differentmethods. By this, a process of suppressing aliasing is performed onlywhen needed, and accordingly, a processing load of the digital stillcamera can be reduced.

Note that, although a case where the user specifies a size of an imagesignal to be output using the operation member 110 has been described asan example, the present invention is not limited to this. When a size ofan image signal obtained at a time of capturing a movie is half a sizeof an image signal obtained at a time of capturing of a still image of aRAW format, the process may automatically proceed from step S1202 tostep S1203 provided that the movie is being captured.

It is apparent that, as a configuration for realizing the noisesuppression process performed by the image processing circuit 105, aconfiguration in which an original image signal is divided into aplurality of frequency components corresponding to different frequencybands which do not overlap with one another may be employed as shown inFIG. 10.

Note that, in the foregoing embodiments, the low-frequency-layer imagesignal is generated by reducing the original signal. However, alow-frequency-layer image signal may be generated by performing an LPFprocess on an original image signal without reducing the image signal.

Furthermore, in any of the embodiments, the different processes areperformed by the different circuits shown in the drawings for simplicityof the description. However, the circuits may be collectively configuredas a single circuit or one of the processes performed by the circuitsmay be performed by some of the circuits.

Other Embodiments

The present invention may be realized by performing a process describedbelow. That is, software (a program) which realizes functions of theforegoing embodiments is supplied to a system or an apparatus through anetwork or one of various storage media, and a computer (or a CPU, anMPU, or the like) included in the system or the apparatus reads andexecutes the program.

As described above, according to the present invention, a noisesuppression process is performed by dividing an image signal accordingto a plurality of bands while and aliasing can be suppressed.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

1. (canceled)
 2. An apparatus configured to generate image dataincluding luminance signal and color difference signals from image datawhich color signal corresponding to a plurality of colors is arranged,the apparatus comprising: a compensation unit configured to performcompensation process to the image data which color signal correspondingto the plurality of colors is arranged, for comprising a plurality ofcolor signals which each pixel corresponds to the plurality of colors; asampling unit configured to generate an image data which that is sampledso that the color signal corresponding to the plurality of colors isarranged, using the image data which the compensation process isperformed; and a luminance signal generation unit configured to generateluminance signal from the sampled image data.
 3. The apparatus accordingto claim 2, further comprising; a noise suppressing unit configured toperform noise suppressing process to image data which the compensationprocess is performed, wherein the sampling unit generates the sampledimage data using the image data which the noise suppressing process isperformed.
 4. The apparatus according to claim 2, wherein thecompensation unit uses the color signal corresponding to the first colorthat the pixel which is located around the focus pixel comprises forcompensating the color signal corresponding to the first color, to thefocus pixel of the image data that does not comprise the color signalcorresponding to the first color among the plurality of colors.
 5. Theapparatus according to claim 2, the sampling unit generates the sampledimage data by sampling color image corresponding to any of the pluralityof colors to each pixels of the image data to which the compensationprocessing is performed.
 6. The apparatus according to claim 2, whereinthe luminance signal generation unit generates first luminance signalusing image data obtained by compensating color signal corresponding tofirst color to pixel that does not comprise color signal correspondingto the first color among the plurality of colors of the sampled imagedata, generates second luminance signal using color signal which isincluded in the sampled image data without distinguishing color signalcorresponding to the plurality of colors, and generates luminance signalwhich synthesizes the first luminance signal and the second luminancesignal or which selects either one.
 7. The apparatus according to claim3, wherein the noise suppressing unit calculates average amount betweencolor signal of focus pixel of image data which the compensation processis performed and color signal which absolute value of differencesbetween color signal of the focus pixel among color signal of aplurality of pixels that is surrounding the focus pixel is underthreshold value, and replaces color signal of the focus pixel by theaverage value.
 8. A method configured to generate image data includingluminance signal and color difference signals from image data whichcolor signal corresponding to a plurality of colors is arranged, themethod comprising: performing compensation process to the image datawhich color signal corresponding to the plurality of colors is arranged,for comprising a plurality of color signals which each pixel correspondsto the plurality of colors; generating an image data which that issampled so that the color signal corresponding to the plurality ofcolors is arranged, using the image data which the compensation processis performed; and generating luminance signal from the sampled imagedata.
 9. The method according to claim 8, further comprising; performingnoise suppressing process to image data which the compensation processis performed, wherein the generating the image data generates thesampled image data using the image data which the noise suppressingprocess is performed.
 10. The method according to claim 8, wherein theperforming compensation process uses the color signal corresponding tothe first color that the pixel which is located around the focus pixelcomprises for compensating the color signal corresponding to the firstcolor, to the focus pixel of the image data that does not comprise thecolor signal corresponding to the first color among the plurality ofcolors.
 11. The method according to claim 8, whether the generating thesampled image data comprises sampling color image corresponding to anyof the plurality of colors to each pixels of the image data to which thecompensation processing is performed.
 12. The method according to claim8, wherein the generating luminance generates first luminance signalusing image data obtained by compensating color signal corresponding tofirst color to pixel that does not comprise color signal correspondingto the first color among the plurality of colors of the sampled imagedata, generates second luminance signal using color signal which isincluded in the sampled image data without distinguishing color signalcorresponding to the plurality of colors, and generates luminance signalwhich synthesizes the first luminance signal and the second luminancesignal or which selects either one.
 13. The method according to claim 9,further comprising calculating average amount between color signal offocus pixel of image data which the compensation process is performedand color signal which absolute value of differences between colorsignal of the focus pixel among color signal of a plurality of pixelsthat is surrounding the focus pixel is under threshold value, andreplaces color signal of the focus pixel by the average value.
 14. Acomputer readable storage medium storing a program for causing acomputer to execute each step of a method defined in claim
 8. 15. Thecomputer readable storage medium according to claim 14, furthercomprising; performing noise suppressing process to image data which thecompensation process is performed, wherein the generating the image datagenerates the sampled image data using the image data which the noisesuppressing process is performed.
 16. The computer readable storagemedium according to claim 14, wherein the performing compensationprocess uses the color signal corresponding to the first color that thepixel which is located around the focus pixel comprises for compensatingthe color signal corresponding to the first color, to the focus pixel ofthe image data that does not comprise the color signal corresponding tothe first color among the plurality of colors.
 17. The computer readablestorage medium according to claim 14, whether the generating the sampledimage data comprises sampling color image corresponding to any of theplurality of colors to each pixels of the image data to which thecompensation processing is performed.
 18. The computer readable storagemedium according to claim 14, wherein the generating luminance generatesfirst luminance signal using image data obtained by compensating colorsignal corresponding to first color to pixel that does not comprisecolor signal corresponding to the first color among the plurality ofcolors of the sampled image data, generates second luminance signalusing color signal which is included in the sampled image data withoutdistinguishing color signal corresponding to the plurality of colors,and generates luminance signal which synthesizes the first luminancesignal and the second luminance signal or which selects either one. 19.The computer readable storage medium according to claim 15, furthercomprising calculating average amount between color signal of focuspixel of image data which the compensation process is performed andcolor signal which absolute value of differences between color signal ofthe focus pixel among color signal of a plurality of pixels that issurrounding the focus pixel is under threshold value, and replaces colorsignal of the focus pixel by the average value.